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Abstract—Massive Open Online Courses (MOOCs) have left 
their mark on the face of education during the recent couple of 
years. At the Hasso Plattner Institute (HPD in Potsdam, Germany, 
we are actively developing a MOOC platform, which provides our 
research with a plethora of e-learning topics, such as learning 
analytics, automated assessment, peer assessment, team-work, 
online proctoring, and gamification. We run several instances of 
this platform. On openHPI, we provide our own courses from 
within the HPI context. Further instances are openSAP, 
openWHO, and mooc.HOUSE, which is the smallest of these 
platforms, targeting customers with a less extensive course 
portfolio. In 2013, we started to work on the gamification of our 
platform. By now, we have implemented about two thirds of the 
features that we initially have evaluated as useful for our purposes. 
About a year ago we activated the implemented gamification 
features on mooc. HOUSE. They have been employed actively in 
the course “Design for Non-Designers”. We plan to activate the 
features on openHPI in the beginning of 2017. The paper at hand 
recaps, examines, and re-evaluates our initial recommendations. 


Keywords— Gamification, e-learning, MOOC, massive open 
online courses 


1. INTRODUCTION 


The term gamification is generally defined as the application 
of game elements in non-gaming contexts [1]. The concept 
remains a fluid term though, as there exists no broadly agreed on 
standard definition yet [2]. Driven by the successful 
implementation of gamification in other contexts such as the 
StackExchange! network, the location-based social network 
Foursquare’, or the Customer Relation Management (CRM) 
software Salesforce’, we started to examine if and how we could 
incorporate game elements in our MOOC platform. Right away 
we found promising gamification examples in the educational 
sector, e.g. Khan Academy*, DuoLingo>, or Codeacademy® to 
name just a few. 


Other researchers have worked on this topic before. Hansch 
et al. [2] for example examined a couple of gamified learning 
platforms including openHPI. For all these platforms, the main 
motive to employ gamification was to increase the engagement 
of their learners. Progress tracking has been named by all 
platforms as the initial step. At least two of the examined 
platforms report that their users keep returning in rather high 
numbers to participate in further courses. Most platforms report 
that while some of their users are very active in the discussion 


forum, the clear majority keeps rather silent. Gamification is 
considered to be a tool to engage participants more intensely 
with the discussion forums in most cases [2]. MOOCs, even 
without gamification, show an inherent tendency to drift towards 
Edutainment [2]. While this is sometimes helpful and 
appreciated, we should be careful not to alienate the serious 
learners. Gamification has, therefore, to be employed very 
carefully. 


We started to investigate the options of gamifying our 
platform back in 2013. In the following year, we worked on 
prototypes addressing several aspects of the gamification of the 
platform. Currently, about two thirds of the features that we had 
evaluated as useful for our purposes during the research phase, 
are implemented in our code base. About a year ago, these 
features went into production and have been activated on 
mooc.HOUSE to evaluate the outcomes. On the other platforms, 
they run in stealth mode, the data collection is activated, but 
there is no user interface that displays the results to the course 
participants. It is planned to reset the collected data and activate 
the features on openHPI in the beginning of 2017. The features 
have been actively promoted in the first iteration of the course 
Design for Non-Designers on mooc.HOUSE. The same course 
had been conducted previously on the same platform, before the 
gamification elements had been deployed. After the deployment 
of the gamification elements, other courses on mooc. HOUSE 
were offered in which they have not been actively promoted by 
the teaching teams. In total this situation provides us with the 
possibility to evaluate the effects of the platform’s gamification 
from different perspectives. In Section II we present the initial 
motivation why to employ gamification and discuss how far 
these thoughts are still valid. Section III discusses some 
theoretical backgrounds. Sections IV to VI present the features 
we have deployed or planned in more detail. Finally, Sections 
VII and VIII present the evaluation of the features and our 
conclusions. 


II]. THEINITIAL PROBLEM SETTING 


One of the criticisms towards MOOCs is that they suffer 
from high dropout rates. When we first started to think about the 
gamification of the platform in 2013, we had an average 
completion rate in our courses of 18.3% (#certificates / 
#participants-at-course-end). Back then this was a very good 
ratio compared to other platforms’. Figure 1 shows how the 
average submission rates of the graded weekly assignments 
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declined throughout the six weeks’ duration of the courses, 
which we offered on openHPI in 2012 and 2013. The 
submissions count for the week | assignment has been defined 
as 100%. The submission rate drops strongly in week 2 and 3 
(82% respectively 73%), but from then on stays rather stable. 
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Figure 1: Average weekly assignment submission rates in the 2012-2013 
courses [3]. 


Although our completion rates already were comparably 
high, we felt the urge to still increase them and advance our 
platform. The idea was to employ gamification to improve the 
completion rates of the courses. Our hope was to keep those 
interested who get bored or lose interest over time by injecting a 
little “extrinsic motivation intervention” here and there. 


We ran the same statistic for the courses that we have 
conducted since then and have similar results. Figure 2 shows 
the submission rates for the courses that have been examined for 
the paper at hand. As in Figure 1, the submissions count for week 
1 has been defined as 100%. Figure 3 shows the passive 
participation in comparison. Here the number of participants that 
have visited more than 10% of week 1’s items have been defined 
as 100%. Both, Figure 2 and Figure 3, show that obviously, the 
gamification features did not have any influence on the results. 


Meanwhile, three years and a couple of courses, surveys and 
user feedback later, our perception of drop-outs has somewhat 
changed. It became obvious that not every participant is 
interested in a certificate. At an extreme, some participants 
might even consider a course a success if they just find a 
particular piece of information in one of the videos. 
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Figure 2: Weekly submission rates in the examined courses. See TABLE III. 
for additional course data. The absolute submissions in week 1 have been: 
1344(BPM2016), 4803(Intsec2016), 705(Javawork2016), 769(DFND 1), 375 
(DFND1-1). The course Industrie4.0 did not have any exams. Gamification 
features have been activated and visible in the course DFND1-1 and 
Industrie4.0. 
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Figure 3: Participants that have accessed more than 10% of the week’s items. 
The number of participants that have accessed more than 10% of week 1’s 
course items have been defined as 100%. Participants that have visited more 
than 10% of the course items in absolute numbers: 3664(BPM2016), 
8670(Intsec2016), 1802(Javawork2016), 1732(DFND1), 1176(DFND1-1), 
5749(Industrie4.0). Gamification features have been activated and visible in 
the course DFND1-1 and Industrie4.0. 


We have very early argued that the discussion on drop-out 
rates is flawed and have already published our findings towards 
these numbers back in 2014 [3]. Based on these thoughts and 
enabled by the more fine-grained data that we have collected by 
now, we changed the way how we calculate the completion 
rates. Currently, only the users that have enrolled before the 
course middle are considered, as they are the only ones who can 
complete the course with a sufficiently good grade. In these 
terms, we have completion rates ranging between 2.8% and 
37.9% in the courses we offered since we moved to the new 
version of our platform in 2014 (see [4] for more detailed 
information on the development of our platform). The average 
completion rate in the examined courses is about 17.75%. In 
comparison, the average would be 16.01% if we use the original 
formula as mentioned on the previous page. The decline 
compared to the results from the 2012 and 2013 courses is 
caused by some outliers. E.g. the course with the lowest 
completion rate (Javaworkshop2015) is a good example why 
even the way we currently calculate the completion rate is still 
not quite correct. This course had a comparably high no-show 
rate (no-shows are people that enroll for a course but never even 
start it), as it had to be postponed for several months due to 
organizational reasons. Renz et al. have already proposed to re- 
calculate the drop-out rates by removing the no-shows from the 
calculation [5]. By following this approach, we achieve more 
accurate completion rates between 5.7% and 42%; 22.8% in 
average. We are dealing with a homemade issue of the MOOC 
community here, as every course provider is eager to show off 
with massive numbers of enrollments. Calculating completion 
rates this way, requires to be honest about the real number of 
participants as well. 


III. EDUCATIONAL BACKGROUND, MOTIVATION, AND 
LEARNER TYPES 


A. A Word of Warning 


Gamification is rooted in the behaviorist education model as 
proposed by B.F. Skinner in the 1950-60s [6] [7]. His 
educational model was built on the theories of J.B. Watson, who 
himself was influenced by Pavlov’s work on conditioned 
reflexes [8]. Skinner proposed that children learn to speak in 
their mother tongue based on a given set of words and sentences, 
positively reinforced by receiving little rewards when they do it 
right [9]. The behaviorist learning theory assumes that all 
behavior are mere reflexes on consequences that result from 
previous behavior. The teacher is assumed to be a benevolent 
dictator rewarding or punishing the learner for right or wrong 
behavior, whereas rewarding correct behavior is rated to be 
superior to punishment [9]. The behaviorist model was very 
dominant in early forms of e-learning. Skinner himself 
developed a teaching machine to be employed in “programmed 
learning.” Questions were shown to students and in the case of 
a correct answer, the student received a reward [10]. E-learning 
often tends to fall back to these primitive mechanisms as they 
are the easiest to implement. Gamification elements, such as 
game points or badges, are a means of positive reinforcement. 
Similar mechanisms are still used in “analogue” learning 
environments, such as “praise cards”, etc. 


Noam Chomsky back in 1959 already doubted that—if 
Skinner was right—humans would be able to learn to use their 
language in its infinite variety of combinations [11] , one of the 
abilities that define the distinction between human and animal. 
We, therefore, should be very thoughtful when we employ 
gamification in a learning context if we do not want to end up 
with a system that favors learning by rote instead of real 
understanding. 


B. MOOCs, MOOC Platforms, and Educational Theory 


The first cMOOCs, provided by George Siemens and 
Stephen Downes, originated from the connectivist theory and 
did not make use of a dedicated MOOC platform at all. The 
following, and in terms of enrollment numbers much larger, 
xMOOCs fell back to a rather classic instructional approach 
where knowledge is imparted by a teacher via the use of video. 
In those days, an ecosystem of MOOC platforms, such as 
Coursera®, Udacity’, edX!°, openHPI!! etc., MOOC search 
engines such as Class Central!?, MOOC list'?, Open Education 
Europa!* and other peripheral resources, such as blogs about 
MOOCs evolved. When edX open sourced their code base many 
other course providers were enabled to run courses on their own 
platforms, based on the edX code, France Université 
Numerique!> or XuetangX'¢ in China to name just a few. 


Some MOOC platforms claim to implement more modern 
educational paradigms. The British platform FutureLearn!’ 
claims to implement a social constructivist model. NovoEd'® and 
openHPI/openSAP/mooc.HOUSE provide increasing support 
for teamwork, etc. The order in which we implemented features 
shows our priorities. CollabSpaces enabling teamwork [12] 
[13], support for practical programming exercises [14] [15] [16] 
or a peer assessment system [17] [13] have had a higher priority 
than the gamification of the platform. That said, we still are 
convinced that, in a few well-defined areas, it will have positive 
effects for our learners in the long run. 


C. Motivation Theory 


In the following, we will briefly present three theories that 
try to answer the question about the motivation of our 
participants to finish a course. For more details see also Staubitz 
et al. [19]. 


In their Self Determination Theory (SDT) [20] Deci and 
Ryan state that people are innately curious and interested 
creatures who possess a natural love of learning and desire to 
internalize knowledge, customs and values. Their term for this 
is intrinsic motivation. They identified three major intrinsic 
motivators: 


e Competence—humans attempt to be efficient and good 
in what they are doing 


e Autonomy—humans attempt to be in command of their 
life 


e Relatedness—humans have a universal desire to interact 
and be connected with others 


Daniel Pink’s Drive Theory [21] differentiates between 
intrinsic and extrinsic motivation. According to him, particularly 
monetary bonuses can be negative and prevent creativity [21]. 
He also identifies three major intrinsic motivators: 


e Autonomy—humans want to make their own choices 


e Mastery—humans attempt to improve what they are 
doing 


e Purpose—humans make 


contributions 


attempt to meaningful 


Andrzej Marczewski combines the insights of the SDT and 
Drive Theory in his RAMP (Relatedness, Autonomy, Mastery, 
Purpose) framework [22]. 


D. User Types 


Marczewski also introduced user types to get rid of the 
overused metaphor of Richard Bartle’s player types in the 
context of gamification [24]. In the 1990s Richard Bartle 
analyzed players of Multi-User Dungeons (MUDs) and 
categorized them into four types—achievers, explorers, 
socializers, and killers—to better understand their behaviors 
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[25]. Marczewski transferred these player types into user types 
for applications in a non-game context. He differentiates four 
intrinsically motivated types—socializers, free spirits, 
achievers, and  philanthropists—plus four  extrinsically 
motivated types—networkers, exploiters, consumers, and self- 
seekers [22] [24]. Marczewski states that different user types 
afford different intrinsic motivation factors. Socializers are 
driven by relatedness, free spirits are driven by autonomy, 
achievers are driven by mastery and philantropists are driven by 
purpose [24]. In Section VII we will examine if we can detect 
any of these user types in the participants’ interaction data in our 
courses. 


Educators often undermine the inherent human tendencies to 
learn and develop by introducing external controls into learning 
climates. Those external controls tend to replace a human’s 
inherent intrinsic motivation with extrinsic motivation. The 
activity is no more performed for its own sake but to obtain some 
separate outcome, usually caused by a third party. This often 
comes in forms of reward or punishment (positive or negative 
reinforcements in behaviorist lingo). Through both, the students’ 
cognition of tasks can change, so tasks which were intrinsically 
fun before are no longer experienced in this way. For example 
supervision, monitoring, and performance evaluation are 
common means in schools which can reduce the original 
motivation of students [26]. 


The concept of gamification mostly increases extrinsic 
motivation by externally creating goals and rewarding 
participants for achieving them. Though rewards are a proven 
way to spur students to put forth effort, this behavior control 
often does not increase the motivation of learning itself but 
rather increases the motivation to achieve the external goal. 
Some educators have refused extrinsic motivational methods 
from the beginning. It can be seen as bribing the students to do 
something they should do anyway because it is in their own or 
in society’s interest. Then again, extrinsic methods can develop 
a minimax mentality, so students do what will bring them the 
most rewards with the least effort [27]. When students become 
aware of being bribed, they start to consider the bribing 
necessary for the activity, as they are not expected to overcome 
it without rewards. The students adopt the view that the activity 
itself is not worth performing in the absence of extrinsic rewards. 
That way, the initially present intrinsic motivation is 
undermined [28]. 


A very powerful yet problematic extrinsic incentive is the 
competition between students. Competitions can be for tangible 
prices or just for the satisfaction of winning (being better than 
the others), between individuals or between groups. They are 
usually structured around test scores or other performance 
measures, to be able to announce a winner. However, although 
it is powerful, Brophy states a number of arguments against the 
application in regular classrooms. To name just a few, students 
tend to focus on the competition rather than on the given task 
and they often do not have the choice if they want to participate 
or not (a lack of autonomy). Furthermore, competition requires 
teams that are balanced by ability profiles, and most of all 
competitions only have one winner but many losers [28]. 
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Figure 4: The new progress page 


IV. GAMIFICATION ROUND ONE 


In 2013 we started to re-model and re-implement our 
platform from scratch (see also [4]). Already in version | of the 
platform, we featured a simple gamification element: progress 
bars. Back then, they have been very simple, only showing the 
percentage of the achieved from the overall available points per 
course section. Typically, our major courses are delivered within 
a timeframe of about six weeks where in general each week 
forms a section. Additional sections are the final exam as well 
as some excursions and additional or related topics. Progress 
bars help users to position themselves within the current course 
by showing what they already have done and what they still need 
to work on. When we moved from version 1 of our platform to 
version 2 (the current one), the options for the gamification of 
the platform were investigated in detail. A group of students 
defined and prototyped different gamification features. Not all 
their suggestions directly made their way to the new platform, 
often due to restraints in terms of time and resources. An 
improved progress overview, however, was directly introduced 
with the launch of the new platform. Instead of just showing the 
mere percentage of achieved points, the user can now see exactly 
how many activities she already has visited and how many 
points she received for this activity (Figure 4). Additionally, we 
animated all the progress bars on this page to fill up from zero 
to the current progress percentage. This emphasizes the “game- 
ish” look-and-feel of the progress page. Unfortunately, we did 
not take any measurements how often the participants visited the 
old progress page, but we do know that the new progress page is 
one of the most viewed pages on the platform. As a future 
improvement, we are working on an even more detailed and 
fine-grained way of displaying the participant’s progress, e.g. by 
showing how many minutes of a video a user has seen. 


At this stage, we also set the foundations for what will 
become the main benefiter of the platform’s gamification in the 
future. The forum plays a very important role on our platform. 
Early on, strengthening the forum discussions and rewarding the 
active participants in the forum was one of the major goals of 
our gamification plans. Figure 5 shows the importance of the 
forum discussions. We can state with great confidence that there 
is a strong correlation between the activity in the forum and the 
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Figure 4: Correlation between completion rate and forum activity 


completion rate of a course (Pearson correlation coefficient: 
0.67). We have presented similar results in previous work [28]. 
Furthermore, the forums play an important role in those courses 
that attempt to take a more social-constructivist approach. In our 
Java courses, we have employed the discussion forum as a “soft” 
exercise type; instead of questions about the content of the most 
recent video, the self-tests in these courses often encourage 
learners to do some research of their own and discuss their 
results in a pre-arranged discussion thread. Also, the hands-on 
exercises often force the participants to go to the forums and ask 
for help or check existing discussions. We will discuss this in 
more detail in a future paper. 


When we developed version 2 of the platform, we added a 
few features to the forums that had been missing in the version 
1. Particularly, the possibility to vote for posts—questions, 
answers, and comments—and the possibility to mark questions 
as answered. These features are prerequisites for some of the 
elements that have been implemented in round two, which will 
be discussed in the following Section. 


V.  GAMIFICATION ROUND TWO 


Due to a variety of reasons it took us quite some time to 
include at least the core set of features in a productive version of 
the platform. We already introduced the intended set of features 
in 2014 [29], it took another year until we started the first 
gamified course on mooc.HOUSE. As this is our smallest 
platform, it was the best choice to verify and evaluate the 
functionality of our newly developed features. Estimated in the 
beginning of 2017, openHPI will follow. 


In detail, the features that we have introduced are badges and 
eXperience Points (XP). Badges are trophies, represented by 
small graphics. They are displayed on the participants’ private 


Figure 5: Badges are available in three variations: yellow, orange, red. These 
colors correspond to gold, silver, and bronze. 


profile pages. It is planned to show these badges on the 
participants’ public profile pages as well; for those who opt-in 
for that option. The decision-making-process on which 
interactions will be rewarded in which way on the platform was 
extensive and included the consultation of an external game 
design expert. The result of this process is shown and explained 
in Tables TABLE I. and TABLE II. 


TABLE I. EXPERIENCE POINTS (XP) FOR COMMUNICATION. 
Activity Explanation XP 
Answering a forum Encourage participants to be active in the 1 
question forum and answer questions, regardless of 
the quality of the answer. 
Answer is accepted Additional points for high quality answers. 30 


by question author Only one answer can be accepted per 
(or teaching team) question. 


Question is up-voted An up-vote on a question indicates either 5 

an interest in the question or an approval of 

the question quality or relevance. In both 

cases, we reward the author. Each 
participant can up-vote a question once. 

Since good questions are likely to be up- 

voted quite often, we give only few points 

per vote. 


User receives an up- Up-votes are quality indicators. In contrast 10 
vote on an answer to questions, quality approval is the only 
motivation that leads to an up-vote action 
for answers. Thus, we can reward it higher 
than an up-vote for a question. 


In addition to the experience points, we added a couple of 
badges, each of them available in three versions: bronze, silver 
and gold as shown in Figure 6. The Communicator badge 
(Figure 6 middle) is available for students that have posted 3 
(bronze), 8 (silver), or 13 (gold) questions, answers or comments 
in the discussion forums. The Knowledgeable badge (Figure 6 
left) will be rewarded to participants, whose answers have been 
accepted by their peers. Furthermore, we have a Self-Tested 
badge (Figure 6 right) for participants that have taken a certain 
number of self-tests. The requirements for the latter badges are 
similar to the ones for the Communicator badge. 


VI. GAMIFICATION ROUND THREE? 


Some of the gamification elements that had been 
recommended in the original paper [29] have still not made their 
way into the platform or will be removed or altered due to our 
findings so far. In this section, we will comment on these 
features, more detailed motivation why we decided to remove or 
alter existing features will follow in Section VII. 


TABLE II. OTHER EXPERIENCE POINTS (XP). 
Activity Explanation XP 


Voluntary self-test 
taken 


We want to encourage the participants to 10 
take the self-tests so they receive as much 

as possible in-time feedback. On the other 

hand, many taken self-tests provide fast 

user feedback to the teaching team: If a 
significant fraction of the participants 

fails in a self-test, it might be worth a 

deeper investigation. 


Continuous 
attendance 


A participant has seen more than 70% of Xx 
the weeks available videos and started a (variable 
self-test or assignment for more than one value) 
week in a row. 

X = #continous_week * 10 

(#continous_week starts with 0, the 

attendance of the first course week is not 

rewarded with any points. Attending the 

first week in a row is rewarded with 10, 

then 20, 30, etc. eXperience Points) 


Since homework submissions are x 
obligatory, they were not chosen for point (variable 
rewards. However, by offering extra value) 
points for early assignment submissions, 
decreasing every day ahead of the 

deadline, we can motivate certain player 

types not to wait until the very last second 

for their submission. 

We offer a reward of n’2 x 5 points for an 

early submission, when a user submits an 
obligatory assignment n days prior to the 

due date, i.e. 180 points for a submission 

6 days earlier, 125 points for 5 days 

earlier, 80 points for 4 days earlier, 

etc. The maximum of points is capped to 

180 (relevant for excursions etc. that run 

longer than one week). 


User submits 
assignment before 
due date 


The most prominent example is the leaderboard. Already in 
the original recommendation, we have been rather critical 
towards this feature. While it might be encouraging for the few 
participants that range at the top of the results, it will demotivate 
the majority of the participants: Having several thousands of 
enrolled participants, most of them are listed with a bad ranking, 
even if they perform well within the course. For this reason, it 
was suggested to integrate a social graph into the platform [19]. 
The social graph allows to create connections between 
participants, which in turn allow to implement social 
leaderboards. Social leaderboards only show comparisons 
between friends. Even if a participant is the last in line, this will 
be way less frustrating, as she will at least always be visible on 
this board on the first page. Even more important in this context 
is that the participants are comparing themselves only to 
participants that they know, so this provides relatedness [19]. 


Before deploying the gamification features to openHPI, we 
will remove the suggested experience points and badges for 
participants that hand-in assignments early. The reason for 
implementing this feature has become technically obsolete. 
When the original paper was written, we still had an issue with 
load peeks right before exam deadlines. Rewarding participants 
with experience points then seemed to be a promising solution 
to flatten the load curve at least a little. By now we have solved 


the problem by means of a better scalable quiz system. We still 
think that encouraging users to hand-in exercises earlier might 
have its benefits. However, we consider this now in the context 
of a personal learning schedule. In future, the user can create her 
own schedule with a new planning tool we are currently 
developing. Encouraging the user to learn according to her plan 
therefore seems desirable, so this can be rewarded as well. 


Finally, we altered the amount of points that are awarded for 
completing voluntary self-tests. More details on this will follow 
in Section VII. 


Our current implementation of the gamification feature also 
still misses the recommended User States from the initial 
proposal. User States are additions to a participant’s display 
name in the discussion forum. The participant will enter a new 
state each time she has passed a certain level of points. The idea 
here is to give other participants a feel for the credibility of the 
peers they are discussing with. This technique is well known 
from diverse expert forums. It took us some time to decide on an 
appropriate scheme. As each instance of the platform has a 
diverse user group and also the different platforms approach 
their users in a different way, it was not easy to find a scheme 
that fits and appeals to all participants. Finally, the decision was 
made to use the kyu ranking system with different “belt” colors 
as used for martial arts. This feature is currently being 
implemented. 


VU. EVALUATION 


A. Point distribution 


We will now make a first attempt to make sense of the data we 
have collected on our platforms. As already mentioned in 
Section I, the preconditions and contexts in which we have been 
collecting the data differ quite a lot between the platforms and 
also between individual courses. While on openHPI the 
gamification feature runs in stealth mode—the users are 
collecting points but the points are not displayed anywhere, the 
gamification feature has been completely activated on 
mooc.HOUSE starting with the course Software Design for Non- 
Designers. While in this course, the participants have been made 
aware of the possibilities to earn experience points and badges, 
in the following course Jndustrie4.0, the users have not been 
actively informed about these possibilities. We selected a 
representative sample of courses on both platforms, which are 
listed in Table II. For the listed courses, we evaluated the 
amount of XP and badges that the participants have gained 
(Figure 7). The data already shows that, in comparison, self-tests 
and early submissions are rewarded extraordinarily high. 
Therefore, we should decrease the amount of points that can be 
gained for these activities while forum activity needs to be 
rewarded more. As the participants, obviously, are already 
intrinsically motivated to solve those kind of tasks, awarding 
them with XPs might be even harmful as we discussed in Section 
III. Motivated by these results we decided to change the rule for 
receiving XPs for submitting self-tests. While the rule that had 
been implemented in round two awards 10 XP for every self-test 
that is submitted by a participant, the updated rule only awards 
2 XP and only for the first 90% (or more) correct submission to 
a self-test. 


TABLE III. 


STATISTICS OF SELECTED COURSES. PARTICIPANTS, NO SHOWS AND RECORD OF ACHIEVEMENTS ARE LISTED. GAMIFICATION POINTS WERE 


COLLECTED IN ALL COURSES. HOWEVER, THEY WERE SHOWN ONLY IN TWO OF THE COURSES. THE AVERAGE FORUM ACTIVITIY (POSTS AND THREADS) PER 
PARTICIPANT WAS CALCULATED USING THE NUMBER OF PARTICIPANTS AT COURSE MIDDLE, NOT INCLUDING NO SHOWS. 


Platform Course Duration _ Participants 
in weeks 
mooc.house Design for Non-Designers 2] 2187 
(Pilot) (DFND1) 
mooc.house Software Design for Non- 6 1305 
Designers (DFND 1-1) 
mooc.house Industrie 4.0 ( Industrie40- 4 6443 
2016) 
openHPI Einfiihrung in die 2 2800 
Testgetriebene Entwicklung mit 
JUnit (Javawork2016) 
openHPI Business Process and Decision 6 5197 
Modeling (BPM2016) 
openHPI Sicherheit im Internet 6 10688 
(Intsec2016) 
XP normalized 
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Figure 6: Gamification points received by participants. The absolute sum of 
points per gamification rule has been divided by the number of points per 
tule item and the number of course participants. Example: We measured a 
total of 1770 XP in Javawork2016 for the rule accepted_answer. This rule 
accepted_answer values for 30 points each. We had a total of 2800 
participants in this course. So we normalized the XPs by applying the 
following formula: measured XP / rule value / # participants (1770 / 30 / 
2800). We finally multiplied this value by 10.000 so that we can display it 
in a diagram with logarithmic scale as otherwise some of the results would 
not be visible at all due to the differences. Where the rule value is 
calculated dynamically, as e.g. for early_submission, we took the average 
between the possible minimal and maximal values. 


No RoA  Gamification Forum Posts Forum Threads 
Shows visible Total (Avg. Per Total (Avg. Per 
User/Week) User/Week) 
n.a. 472 No 350 (0.03) 97 (0.01) 
na 167 Yes 530 (0.07) 254 (0.03) 
na. na Yes 658 (0.03) 212 (0.01) 
938 284 No 814 (0.15) 143 (0.04) 
1724 851 No 1717 (0.06) 403 (0.05) 
1850 3482 No 7788 (0.12) 1120 (0.01) 


B. Detecting User types 


A new feature of our learning analytics engine enabled us 
to cluster participants by their interaction with the platform. 
We attempted to determine if we can separate our participants 
into socializers, achievers, and explorers—three of the user 
types defined by Marczewski. While the clustering itself is not 
really helpful here, the statistical and graphical output of this 
feature provides several interesting insights. We’re capturing 
our analytics data similar to the TinCanAPI? format in the 
form of “Noun, verb, object”. Some of the verbs are then 
combined, according to rules that we have defined, to metrics 
(see also [30]). In the following, we will examine some of 
these verbs and metrics: 


e = course_performance(CP): a value between 0 and 1. 
Shows the percentage of points a participant has 
received in a course. Includes all main and bonus 
assignments”°. 


e =item_discovery(ID): the amount of items that a 
participant has seen in a course. The amount of 
available items differs from course to course. 


e =forum_activity(FA): a metric calculated as a weighted 
sum of textual_forum_contribution and forum_obser- 
vation, whereas textual_forum_contribution has a 5- 
times higher weight than forum_observation. 


e = forum_observation(FO): the amount of question 
visits and subscriptions of a participant in a forum 


'9 http://tincanapi.com/overview/ 

20 Tn terms of the way how achieved points are included into the 
course result, our courses feature three types of assignments. Self- 
tests, main assignments, and bonus assignments. Self-tests are not 


included at all in the course result, main assignments define the 
amount of possible points in a course, and bonus assignments 
allow to fill gaps in the main assignments. 


e = textual_forum_contribution(TFC): the amount of 
questions, discussions, answers, or comments that a 
participant has contributed. 


We ran the clustering on the three openHPI courses that 
are listed in TABLE II. We could not run the same analysis 
on mooc.HOUSE as the clustering feature is not yet activated 
there. Figures 8, 9, and 10 show these clustering attempts in 
the the three listed courses. The colors in the scatterplots 
represent the clusters that have been formed by our clustering 
mechanism and does not contain relevant information in our 
context. Next to the scatterplots, the clusterer also generates a 
correlation matrix, which is shown in TABLE IV. From the 
correlation matrix and the scatterplots we derive the following 
statements: 


There is only a very weak correlation between a 
participant’s forum activity (FA) and her performance (CP). 
This, combined with our finding that there is a strong 
correlation between the total forum activity in a course and the 
course completion rate, indicates that there might exist a group 
of participants that fits the socializer definition. 


We will now compare the correlations between forum 
contribution (TFC), forum observation (FO), and forum 
activity (FA) in the three evaluated courses. In Javawork2016 
all three metrics are strongly correlated. In BPM2016, only the 
forum observation has a strong correlation to the forum 
activity. In Intsec2016, particularly the very weak correlation 
between forum contribution and forum observation is 
interesting. In other words: in Javawork2016, the participants 
were communicating with each other and the teaching team, 
while in BPM2016 the participants mostly read the forum. In 
the Intsec2016, the participants posted a lot (see also TABLE 
III. ), but hardly any of the posters also read the forum. 


TABLEIV. | CORRELATTIONS MATRIX FoR FoRUM_ACTIVITY (FA), 
TEXTUAL_FORUM_CONTRIBUTION (TFC), FORUM_OBSERVATION (FO), 
COURSE PERFORMANCE (CP), AND ITEM DISCOVERY (ID). COLOUR CODE 
-- DARK RED: VERY STRONG CORRELATION, RED: STRONG CORRELATION, 
ORANGE: MODERATE CORRELATION, YELLOW: WEAK CORRELATION, 
GREY: VERY WEAK CORRELATION. 


Course FA TFC FO CP ID 
Javawork FA 0.722 0.991 


2016 TFC 1 0.621 
FO 
CP 
ID 
BPM FA 
2016 TFC 
FO 
CP 
ID 
Intsec FA 
2016 


We can also see that the Java workshop—even if the 
correlation is still weak—shows a significantly higher 
correlation than the other courses between forum activity and 
course performance. 


If we have a closer look at the scatterplots in Figure 8-10, 
we can find a group of participants, in each of the examined 
courses, that has visited many of the course items, but did not 
perform well enough to be eligible for a certificate (Figure 8 
— 10, top — 1). In other words: they are exploring the platform. 
Close enough to Bartle’s definition of explorers or 
Marczewski’s definition of free spirits. Then there is a group 
of participants that does not seem to consume more items than 
absolutely necessary to get a good result: achievers (see 
Figure 8 — 10, top — 2). Another interesting observation is 
shown in Figure 8 — 10, bottom — 3. These are participants, 
that performed well in all the exams they took, but in total they 
were not eligible for a certificate. These are what we might 
call the drop-outs (maybe a couple of drop-ins as well). In all 
three cases, it does not matter where exactly we draw the 
borders as they only determine the size of the group but not its 
existence. 
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Figure 7: An attempt to cluster separate user types in the course Einfiihrung 
in die Testgetriebene Entwicklung mit JUnit (Javawork2016). In this and 
the following figures, the left and the right image display two different 
dimensions of the clustering. 
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Figure 8: An attempt to cluster separate user types in the course Business 
Process and Decision Modeling (BPM2016). 
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Figure 9: An attempt to cluster separate user types in the course Sicherheit 
im Internet (Intsec2016). 


VUI. FUTURE WORK 


As already mentioned, the implemented gamification 
features will be deployed to the openHPI platform where they 
also will be promoted more prominently. As our intention is 
to reduce the behavioristic character of gamification and 
rather push it into a more social constructivist direction, we 
will avoid to put too much stress on the competitive character 
of the feature and strengthen the collaborative part. 
Particularly, our long planned social graph feature will play a 
major role here. We already have experimented with 
mentioning the most supportive participants in a few courses. 
We plan to do this on a more regular basis to value the 
contribution of these participants and motivate more people to 
actively participate. In addition, we think about offering those 
people to become kind of a moderator for future courses to 
help the teaching team with supporting the forums as well as 
to positively influence the atmosphere within the course. 


IX. CONCLUSION 


Employing gamification on an e-learning platform must 
be handled very carefully. A thoughtless implementation will 
let the platform fall back to the simple positive reinforcement 
mechanisms of behaviorism and programmed learning which 
are not what we desire. Our decision was, therefore, to 
strengthen the collaborative and communicative aspects of 
gamification by subtly gamifying our main communication 
channel: the discussion forum. This enables us not only to 
award the efforts of our learners that are active in the forum, 
but also to employ the discussion forum as a sort of “soft” 
exercise type. Although the active participants are not honored 
with a better course grade, gamifying the forum offers at least 
a lightweight appreciation for their work. We have shown that 
our interaction data supports the hypothesis that a user group 
that fits Marczewski’s definition of socializers exists. We 
have also shown that this group benefits the success rate of the 
course in total and thus play an important role in the courses. 
These very important activities within the courses are 
currently not duly rewarded, which is one of the reasons why 
we decided to focus our gamification efforts on this particular 
area of the platform. With our approach of rewarding high 
quality discussions in the forums, we attempt to bend 
gamification from its rooting in behaviorism into a more 
social constructivist direction. It is in the nature of the question 
that it is harder to separate explorers from achievers. We were 
able to show, however, that at least a group is existing that is 
exploring the platform features without being high 
performers. The question that is still to be answered is how far 
this group can (or should) be motivated to perform better. Or 
how far it makes sense to provide this group of participants 
with a better usage experience of our platform where our main 
goal still should be to enable those participants who want to 
learn to learn what they want. 
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